Skip to content

feat: Add computer use tools for remote and local GUI automation #359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
edenreich wants to merge 14 commits into
base: main
Choose a base branch
from
Open

feat: Add computer use tools for remote and local GUI automation #359

edenreich wants to merge 14 commits into from

Conversation

@edenreich
Copy link
Contributor

@edenreich edenreich commented Jan 2, 2026

Adds computer use capabilities including screenshot capture, mouse movement, mouse clicks, and keyboard typing. Includes a complete Docker-based example with Ubuntu GUI desktop, X11/Wayland support, and live screenshot streaming to web UI. Closes #358.

Key features:

  • Screenshot tool with streaming support
  • Mouse control (move and click)
  • Keyboard input (text and key combos)
  • Rate limiting and approval system
  • Complete Docker example with web UI integration

Technical details:

  • Supports both X11 and Wayland display servers
  • Screenshot streaming via WebSocket for live desktop viewing
  • Circular buffer for efficient screenshot storage
  • Rate limiting to prevent abuse
  • User approval system for sensitive operations
  • Docker example with headless Ubuntu desktop setup

@edenreich
feat: Add computer use tools with screenshot streaming and remote GUI…
ebe5cef 
… control
@edenreich
refactor(web): Rename screenshot overlay to preview
841bb09 
Rename screenshot streaming UI components to use "Preview" terminology:
- Rename screenshot-overlay.js to preview-overlay.js
- Remove emoji from button (📷 Screenshots → Preview)
- Update overlay title from "Live Screenshot" to "Live Preview"
- Update user-facing messages to use "Preview" instead of "Screenshot"

This improves clarity and consistency in the web UI while keeping
internal implementation details (CSS classes, API endpoints) unchanged.

Signed-off-by: Eden Reich <[email protected]>
@edenreich
feat: Add typing delay and improve X11 keyboard support
5bd5520
@edenreich
refactor: Refactor computer use tools with display protocol abstraction
9c32649 
Signed-off-by: Eden Reich <[email protected]>
@edenreich
docs: Add computer use tools documentation
ab05f03
@edenreich
feat: Add macOS support and focus management tools
0153abc
@edenreich
chore: Enable computer use
69ead75
@edenreich
Copy link
Contributor Author

edenreich commented Jan 4, 2026 •
edited
Loading

TODOs

  • Check whether it's a good idea to replace the switch from terminal to active window and back with a GUI window that is always on top - similar to how vercept AI is doing it and only show this when computer use is enabled and it's not a remote session over pty - basically only for local computer use it's necessary because the window is constantly changing focus
  • I should probably also have a visual indicator that the computer is currently watched when computer_use.screenshot.streaming is enabled

@edenreich edenreich changed the title feat: Add computer use tools for remote GUI automation feat: Add computer use tools for remote and local GUI automation Jan 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[FEATURE] Add tools for computer use

2 participants

@edenreich